Automating the Detection of Anomalies and Trends from Text

Scalable and robust nonnegative matrix factorization (NMF) algorithms and software are needed for the generation of feature vectors from text corpora. By preserving nonnegativity, the NMF facilitates a sum-of-parts representation of the underlying term usage patterns in textual data. Both training and test sets of documents can be parsed and then factored by the NMF to produce a reduced-rank representation of an entire document space. The resulting feature and coefficient matrix factors are then used to cluster documents. Recent studies with documents from the Aviation Safety Reporting System (ASRS) have shown that (known) anomalies of training documents can be directly mapped to NMF-generated feature vectors. Dominant features (tracking words or sentences) of test documents can then be used to generate anomaly relevance scores for those documents.